Systems and Methods For Haptic Confirmation Of Commands

ABSTRACT

Systems and methods for haptic confirmation of commands are disclosed. For example a system for generating haptic effects to confirm receipt of a voice command includes a microphone; a housing configured to be contacted by a user, and an actuator in communication with the housing, the actuator configured to output a haptic effect to the housing. The system also includes a processor in communication with the microphone and the actuator, the processor configured to receive speech information from the microphone; recognize the speech information and determine a command associated with the speech information. If the speech information is recognized and the command is determined, the processor is configured to generate a first actuator signal configured to cause the actuator to output a first haptic effect, and transmit the first actuator signal to the actuator. Otherwise, the processor is configured generate a second actuator signal configured to cause the actuator to output a second haptic effect; and transmit the second actuator signal to the actuator.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 13/603,578, entitled “Systems and Methods for HapticConfirmation of Commands,” filed Sep. 5, 2012, which is a continuationof U.S. patent application Ser. No. 12/612,230, entitled “System andMethods for Haptic Confirmation of Commands,” filed Nov. 4, 2009, theentirety of both of which is hereby incorporated by reference herein.

FIELD OF THE INVENTION

The present invention generally relates to haptic feedback and moreparticularly to haptic confirmation of commands.

BACKGROUND

Commands to electronic devices have typically been issued by pressing abutton or flipping a switch. However, voice and other types of commandsare becoming more prevalent in user interfaces, such as thevoice-commanded dialing of cell phones. In such systems, a user mayspeak a command into a microphone to dial a friend's phone number, andthe user may hear a beep or see a light flash to indicate that the phoneis dialing the number. But if the cell phone's speaker is already beingused, such as because the phone is playing a song, the cell phone mayinterrupt the music to play the beep, or, if sounds have been muted, itmay not provide an acknowledgement to the user at all. Thus, it may bedesirable to provide other mechanisms for providing responses to theuser.

SUMMARY

Embodiments of the present invention provide systems and methods forhaptic confirmation of voice commands. For example, in one embodiment, amethod for haptic confirmation of commands comprises receiving speechinformation from a microphone, recognizing the speech information anddetermining a command associated with the speech information, and if thespeech information is recognized and the command is determined,generating a first actuator signal configured to cause an actuator tooutput a first haptic effect, and transmitting the first actuator signalto the actuator. Otherwise, generating a second actuator signalconfigured to cause the actuator to output a second haptic effect, andtransmitting the second actuator signal to the actuator. Anotherembodiment comprises a computer-readable medium comprising program codefor executing such a method.

These illustrative embodiments are mentioned not to limit or define theinvention, but to provide examples to aid understanding thereof.Illustrative embodiments are discussed in the Detailed Description, andfurther description of the invention is provided therein. Advantagesoffered by various embodiments of this invention may be furtherunderstood by examining this specification.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentinvention are better understood when the following Detailed Descriptionis read with reference to the accompanying figures, wherein:

FIGS. 1-4 show block diagrams of systems for haptic confirmation ofcommands according to embodiments of the present invention;

FIG. 5 shows a flow diagram of a computer-implemented method for hapticconfirmation of commands according to one embodiment of the presentinvention;

FIG. 6 shows a block diagram of a system for haptic confirmation ofcommands according to embodiments of the present invention; and

FIG. 7 shows a flow diagram of a computer-implemented method for hapticconfirmation of commands according to one embodiment of the presentinvention.

DETAILED DESCRIPTION

Embodiments of the present invention provide systems and methods forhaptic confirmation of commands. For example, in one embodiment, a usermay speak a command into a microphone on a cellular telephone, such as“call home.” The phone attempts to recognize the spoken command, and ifit is successful, it generates a haptic effect to provide a tactileindication to the user that the spoken command was recognized and thatthe phone will perform the requested function. The phone may thenattempt to make a phone call to a telephone number corresponding to“home.” However, if the phone does not recognize the command, itgenerates a second haptic effect to provide a tactile indication to theuser that the spoken command was not recognized and that no functionwould be executed.

In another embodiment, a first user and a second user may be incommunication with each other using wireless communication devices, suchas radios. Each user has a headset with a microphone and a radio incommunication with the microphone. The headset also includes an actuatorfor outputting haptic effects. The first user may issue a verbal commandto the second user over the radio. The second user may be unable toprovide a verbal response to the first user. For example, the seconduser may be located in a combat zone near enemy combatants. The seconduser may respond to the verbal command by pressing a button or series ofbuttons on his head set or on the radio to cause a haptic message to betransmitted to the first user. The first user's radio may receive thehaptic message, and transmit a signal to the actuator in the firstuser's headset to cause the actuator to output a haptic effect to thefirst user.

Illustrative System for Haptic Confirmation of Commands

Referring now to the figures in which like numbers refer to likeelements throughout the several drawings, FIG. 1 shows a block diagramof a system for haptic confirmation of commands according to oneembodiment of the present invention. In the embodiment shown in FIG. 1,the system comprises a device 100 having a housing 105. The devicefurther comprises a processor 110, a memory 115, an actuator 130, and amicrophone 135, each of which is disposed in, or coupled to, the housing105. In the embodiment shown in FIG. 1, the device 100 is a cell phoneand the housing 105 is configured to be grasped by a user, who can speakinto the microphone 135. In some embodiments, the device 100 may be aheadset comprising a housing 105, a processor 110, a memory 115, anactuator 130, and a microphone 135. In some embodiments, the device 100may comprise a system having a plurality of devices. For example, FIG.2, which will be discussed in greater detail below, comprises twodevices 110, 150 in communication with each other.

Referring again to the embodiment shown in FIG. 1, the processor 110 isin communication with the memory 115, actuator 130, and microphone 135.The microphone 135 encodes speech information received from the userinto one or more microphone signals that are transmitted to theprocessor 110. The processor 110 receives the microphone signals andexecutes voice recognition software stored in memory 115 to attempt torecognize the speech information encoded within the received microphonesignals. If the processor 110 is able to recognize the speechinformation, it executes software stored in memory 115 that attempts toidentify a command associated with the recognized speech information. Ifthe processor 110 identifies a command associated with the recognizedspeech information, it generates a first actuator signal configured tocause the actuator to output a first haptic effect. The first hapticeffect is configured to indicate to the user that the voice command wasrecognized. The processor 110 then transmits the first actuator signalto the actuator 130, which outputs the first haptic effect based on thefirst actuator signal.

However, if the speech information is not recognized, or if a commandcorresponding to the speech information is not found, the processor 110generates a second actuator signal configured to cause the actuator 130to output a second haptic effect. The processor 110 then transmits thesecond actuator signal to the actuator 130, which outputs the secondhaptic effect based on the second actuator signal. The second hapticeffect is configured to indicate that the speech information was notrecognized, or that a corresponding command was not found. In oneembodiment, however, different haptic effects may be output for afailure to recognize the speech information or failure to find a commandcorresponding to the recognized speech information. Note that theidentifiers “first” and “second” are used here to differentiate betweendifferent signals and effects, rather than a sequence of particularsignals or effects. For example, if the voice command is not recognized,only the second actuator signal is generated and transmitted to theactuator; the first signal, indicating that the voice command wasrecognized, is not generated or transmitted to the actuator.

The embodiment in FIG. 1 can comprise any of a number of devices, suchas a handheld device, a wearable device, a vehicle (e.g. a car, awheelchair, etc.), a non-portable device (e.g. a desktop computer), orother devices capable of receiving voice commands, processing them, andgenerating haptic effects.

For example, in one embodiment, the device 100 comprises a personaldigital assistant (PDA). In such an embodiment, a user may speak a voicecommand into the microphone 135 to cause the PDA to perform a function.For example, the user may instruct the PDA to add an appointment to thecalendar for Tuesday at 3 pm. The processor 110 attempts to recognizethe voice command and if the processor 110 recognizes the voice command,the processor 110 then generates an actuator signal and transmits theactuator signal to the actuator to cause the actuator to output a hapticeffect to indicate that the voice command was recognized. Alternatively,if the voice command was not recognized, the processor 110 may generatea second actuator signal and transmit the second actuator signal to theactuator to cause the actuator to output a haptic effect to indicatethat the voice command was not recognized.

In a similar embodiment, the processor 110 may partially recognize thecommand. For example, the processor 110 may recognize that the speechcommand was to add an appointment to the calendar, but may not haverecognized the time of the appointment. In such an embodiment, theprocessor 110 may generate a third actuator signal and transmit thethird actuator signal to the actuator to cause the actuator to output ahaptic effect to indicate that the voice command was partiallyrecognized. In such an embodiment, the haptic effect may indicate notonly that the command was partially recognized, but which part of thecommand was not recognized. For example, if the date was not recognized,the device may output a fourth haptic effect, while if the time was notrecognized, the device may output a fifth haptic effect. Using hapticsignaling, a user may be apprised of difficulties understanding thespoken command without needing to resort to viewing a display todetermine the source of the problem.

In a similar embodiment, the processor 110 may generate actuator signalsas parts of a speech command are received and recognized or notrecognized. For example, in one embodiment, a user may say “addappointment to calendar for Tuesday, September 2 at 3 pm.” The processormay recognize “add appointment to calendar” and generate a firstactuator signal to cause the actuator to output a first haptic effect toindicate that a portion of the command was recognized. The first hapticeffect is then output very shortly after the user has said “addappointment to calendar” to indicate to the user that this portion ofthe speech command was recognized. Another haptic effect may then beoutput after the date is recognized or not recognized, and a thirdhaptic effect may then be output after the time is recognized or notrecognized. The user may then re-state the unrecognized portion of thecommand. Haptic effects that are output corresponding to parts of aspeech command may indicate to the user which portion of the commandneeds to be repeated rather than requiring the user to restate theentire command.

Still further haptic effects may be output to a user in response to avoice command. For example, in the embodiments described above in whicha user attempts to add an appointment to a calendar on a PDA, additionalhaptic effects may be used. For example, if the processor 110 recognizesthe command, but determines that the commanded date and time wouldpresent a conflict with an appointment already stored in the calendar,the processor 110 may generate an actuator signal to cause the actuatorto output a haptic effect to indicate that the command was recognized,but that a potential conflict exists. In a similar embodiment, if theprocessor 110 recognizes the command, but determines that the commandeddate and time are adjacent to an existing appointment, the processor 110may generate an actuator signal to cause the actuator to output a hapticeffect to indicate that the command was recognized, but that the desireddate and time is adjacent to an existing appointment. In such anembodiment, the haptic effect indicating a potential conflict isdifferent than the haptic effect indicating that the appointment time isadjacent to another appointment.

As described above, some embodiments of the present invention mayattempt to recognize speech information as it is received, such as inreal-time. However, even in some embodiments that do not attempt torecognize speech information as it is received, it may be advantageousto perform other functions when speech information is received. Forexample, in one embodiment, a device 100 receives speech informationfrom the microphone and attempts to recognize the speech information asit is received. As the device 100 begins to recognize the speechinformation, it may recognize a word or phrase that corresponds to acommand. For example, if the user says “call home,” the device 100 mayrecognize the word “call” and determine that it corresponds to a commandto start a phone call. This may indicate to the device that the user isissuing a voice command, and the device may prepare to output a hapticeffect before receiving the entire command. For example, the device 100may warm up an amplifier (not shown) that supplies current to theactuator 130 in anticipation of generating haptic effect. The device 100may select one or more potential haptic effects that may be output,depending on what additional speech information may be received. In oneembodiment in which the device 100 does not attempt to recognize speechinformation as it is received, the device 100 may begin to warm-up anamplifier when it detects that it is receiving speech information. Suchan embodiment may determine that speech information is being receivedand the speech information may comprise a voice command. Thus, it may beadvantageous for the device 100 to warm-up the amplifier while it isreceiving the speech information to ensure a haptic effect may be outputshortly after a voice command has been recognized or not recognized.

In addition to handheld embodiments, other devices may provide hapticconfirmation of voice commands. For example, in one embodiment, thedevice 100 comprises a desktop computer. A user may issue a command tocause the computer 100 to perform a function that takes an extendedperiod of time to complete, such as to execute a software build. If theprocessor 110 recognizes the command, it may generate a first actuatorsignal configured to cause the actuator 130 to output a haptic effect tothe user, such as through an actuator 130 embedded within a keyboard ormouse (not shown) coupled to the computer 100. Then, as the processor110 executes the function, it may periodically transmit a secondactuator signal to the actuator 130 within the keyboard or mouse toindicate that the function is still executing. Such a haptic effect maybe a low magnitude vibration designed to provide unobtrusive statusinformation to the user. In another embodiment, the vibration mayinitially be unobtrusive, but may get stronger as the function getsnearer to completion. For example, the magnitude of the vibration mayincrease, or the frequency may increase or decrease. When the processor110 completes the function, it may generate and transmit a thirdactuator signal to the actuator 130 within the keyboard or mouse toindicate that the function has completed. Alternatively, if an erroroccurs during execution of the function, such as a build error, a hapticeffect may be output to indicate that the function terminatedunsuccessfully. Such an embodiment may be useful in contexts where auser begins a function on a device 100 and allows the function toexecute in the background, but the device 100 is able to keep the userapprised of the status of the function. For example, a user may issue avoice command to his cell phone to initiate a download of an applicationfrom a remote server over a cellular network. The user may then put thecell phone back in his pocket, initiate a phone call, or perform someother task. However, the cell phone may still be able to provide hapticstatus of the download to the user, such as percentage of the downloadcompleted, a problem with the download, or a haptic indication when thedownload has completed.

Referring now to FIG. 2, FIG. 2 shows a block diagram of a system 200for haptic confirmation of commands according to one embodiment of thepresent invention. The system 200 shown in FIG. 2 comprises a firstdevice 100 and a second device 150 where each device 100, 150 is incommunication with the other device using connection 170. The firstdevice 100 comprises the components described with respect to FIG. 1.The second device comprises a processor 160 in communication with amemory 165. The processor 110 in the first device 140 is incommunication with the processor 160 in the second device 150. Forexample, in one embodiment, the two processors 110, 160 communicate overa wireless connection, such as via Bluetooth. In some embodiments, thewireless connection may comprise an infrared link, an RF link, a Wifilink, or other wireless connection. In another embodiment, the twoprocessors 110, 160 are in communication over a wired connection, suchas a serial or parallel connection or Ethernet connection. Using theconnection 170, processor 110 is able to transmit signals to and receivesignals from processor 160, though in some embodiments the connection170 may be a one-way connection.

For example, in one embodiment, device 100 comprises a Bluetooth headsetand device 150 comprises a cell phone. In such an embodiment, Themicrophone 135 encodes speech information received from the user intoone or more microphone signals that are transmitted to the processor110. The processor 110 in the headset then causes the microphone signalsto be transmitted to the processor 160 in the cell phone. Processor 160then executes voice recognition software stored in memory 165 to attemptto recognize the speech information encoded within the receivedmicrophone signals. If the processor 160 is able to recognize the speechinformation, it executes software stored in memory 165 that attempts toidentify a command associated with the recognized speech information. Ifthe processor 160 identifies a command associated with the recognizedspeech information, it generates a first actuator signal configured tocause the actuator to output a first haptic effect. The processor 160transmits the first actuator signal to processor 110, which thentransmits the first actuator signal to the actuator 130. In someembodiments, the processor 110 transmits a signal to the actuator tocause the actuator to power-up in preparation for outputting a hapticeffect. In another embodiment, processor 160 transmits the firstactuator signal to the actuator 130.

However, if the speech information or a corresponding command is notrecognized, the processor 160 generates a second actuator signalconfigured to cause the actuator 130 to output a second haptic effect.The processor 160 transmits the second actuator signal to processor 110,which then transmits the second actuator signal to the actuator 130. Inanother embodiment, processor 160 transmits the second actuator signalto the actuator 130. After receiving the second actuator signal, theactuator 130 outputs a second haptic effect based on the second actuatorsignal.

In some embodiments, processor 160 may transmit high level actuatorsignals to processor 110. Processor 110 may then generate a low-levelactuator signal based on the high-level actuator signal. Processor 110may then transmit the low-level actuator signal to the actuator 130 tocause the actuator 130 to output a haptic effect. More detaileddescriptions of high-level and low-level actuator signals may be foundin U.S. Pat. No. 5,734,373, entitled “Method and Apparatus forControlling Force Feedback Interface Systems Utilizing a Host Computer,”filed Dec. 1, 1995 and issued Mar. 31, 1998, the entirety of which isincorporated herein by reference.

Referring again to FIG. 2, one embodiment of the present invention mayprovide haptic effects that emulate the command determined by theprocessor 110, 160 or the function performed by the processor 110, 160.For example, in one embodiment, the first device 100 comprises an earbudand the second device 150 comprises a cell phone. In such an embodiment,a user may issue a voice command, such as “call home.” The processor 110transmits the speech information received from the microphone 135 to theprocessor 160 in the second device 150. The processor 160 recognizes thevoice command and generates a series of haptic effects corresponding toeach number dialed and each ring of the phone being called. Actuatorsignals corresponding to the haptic effects are transmitted to theprocessor 110 in the first device 100, which outputs the haptic effects,such as at a time corresponding to each button press and each ring.

In other embodiments, a haptic command may be received from a processorassociated with a remote device. For example, FIG. 3 shows a blockdiagram of a system 300 for haptic confirmation of commands according toone embodiment of the present invention. In the embodiment shown in FIG.3, a system 300 for haptic confirmation of commands comprises ahaptically-enabled medical alert device 100 in communication with acommunications device 150. The communications device is in communicationwith a remote system 170. In this embodiment, the medical alert device100 is in communication with the communication device 150 using an RFlink, such as an RF link using standard cordless phone frequencies. Themedical alert device 100 is configured to be worn by a user, such asaround the user's neck or wrist. The medical alert device 100 can beused by the user to indicate a medical emergency, such as a fall orheart attack. For example, the user may speak into a microphone on thedevice, such as by saying “help” or by pressing a button and speaking acommand, such as “help” or “emergency.” In the embodiment shown in FIG.3, the processor 110 in the medical alert device 100 then transmits oneor more signals to the processor 160 in the communications device 150.The communications device 150 then initiates a connection to the remotesystem 170 and transmits one or more signals to the remote system 170indicating that the user has signaled a medical emergency. In such anembodiment, the remote system 170 may be monitored by a dispatcher whocan dispatch an ambulance or other services to aid the user.

During the transmission of messages between the medical alert device100, the communication device 150, and the remote system 170, additionalhaptic effects may be generated and output. For example, in theembodiment shown, the communication device 150 may send a message to themedical alert device 100 that causes the medical alert device 100 tooutput a haptic effect to indicate that an emergency call has beeninitiated. After the call has been answered, the communication device150 may cause a second haptic effect to be output to indicate that thecall was answered. Alternatively, or in addition, the dispatcher maycause a signal to be transmitted across the network 380 to the processor160 in the communications device 150. For example, the dispatcher canprovide haptic signals to the user to indicate that their request wasreceived and that help is on the way. The processor 160 may thentransmit a high-level actuator signal to the processor 110 in themedical alert device to cause a haptic effect to be output. Theprocessor 110 in the medical alert device may then generate a low-levelactuator signal based on the received signal from the processor 160 andtransmit the actuator signal to the actuator to cause a haptic effect tobe output. For example, the dispatcher may transmit a long vibration toindicate that the request was received, followed later by additionalvibrations to indicate that an ambulance has been dispatched. Thedispatcher may periodically transmit such haptic signals to indicatethat he is still paying attention to the user and their request is beingresponded to. Such an embodiment may be advantageous for a person whohas difficulty hearing or seeing. It may also provide assurance to theuser that their request for help has been received and that thedispatcher is still handling their needs.

Referring now to FIG. 4, FIG. 4 shows a block diagram of a system 400for haptic confirmation of commands according to one embodiment of thepresent invention. The system 400 shown in FIG. 4 comprises two devices410, 412 in communication with each other via network 480. Device 410comprises a device 100 as shown in FIG. 1. Device 425 also comprises aprocessor 430, a memory 115, a microphone 435, and an actuator 440 asdescribed with respect to FIG. 1. However, each device 410, 412 alsocomprises a network interface 315. Each network interface 315 is incommunication with its respective processor 110, 430 and is configuredto communicate with network 480, thus enabling the devices 410, 412 tocommunicate with each other over network 480. In such an embodiment,device 410 is capable of receiving a voice command from its microphone435 and transmitting the voice command to device 412. For example, inone embodiment, the speech information received from the microphone 135may be transmitted over the network 480 so that a user of device 412 canhear the voice command, such as via a speaker (not shown). The user mayconfirm that the command was recognized by pressing a button on device425 to send a signal indicating an acknowledgment that the voice commandwas received. Device 410 may receive the signal, and processor 110 maygenerate an actuator signal and transmit the actuator signal to theactuator to cause the actuator to output a haptic effect to the user. Insome embodiments, the user may respond by providing a haptic indicationthat the command was not received, or that the command was garbled orunintelligible. In one such embodiment, the user may also provide ahaptic indication to request a re-transmission of the message.

In another embodiment, a user may respond to a received voice command byproviding haptic inputs, such as by pressing a button or tapping on atouch-sensitive device. Each of the haptic inputs may be correlated to aresponse, which may be converted to a spoken command and transmitted toa recipient. For example, a soldier may tap a response to a command intoa touch-sensitive input device coupled to a communication device, suchas device 412, in a sequence to indicate that the command was receivedand that the soldier will comply. The device 412 may then transmit thetactile response to another device, such as device 410, which thenconverts the tactile response into an audible signal, such as a spokenmessage. In such an embodiment, the tactile response may correspond to apre-recorded message or to pre-determined text that may be converted tospeech by the device 412. Such an embodiment, would allow for a silentacknowledgement by a user that results in a spoken reply to therecipient.

The system shown in FIG. 4 may also be advantageously used by multipleusers to perform coordinated tasks or procedures. For example, in oneembodiment, devices 410 and 412 may comprise computers in differentlocations within a bank. Each computer 410, 412 is in communication witha security system that controls access to a bank vault. Users at the twocomputers may need to perform functions, or provide authentication toopen the bank vault. For example, a user at the first computer 410 maybe required to speak a command, such as a word or phrase, to verifyauthorization to open the vault. After the command is recognized, ahaptic confirmation may be sent to a second user at the second computer412 to indicate that they need to speak a command, such as another wordor phrase, to complete authentication to open the vault. If the seconduser's command is recognized, a haptic confirmation may be output ateach of the two devices 410, 412 to indicate to the users that thecommand was successful. At such a time, the lock on the bank vault maybe disengaged.

However, if one of the commands was not recognized, one or both usersmay receive a haptic indication that the authentication failed. In oneembodiment, the user whose command failed may be given a secondopportunity to provide authentication. In such an embodiment, a hapticeffect indicating that the authentication failed but that the commandmay be retried may be output to one or both users. However, if theauthentication has finally failed such that no retries are available, adifferent haptic effect may be output to one or both users to indicatethat authentication failed.

Still further embodiments including different or more complicatedsequences of events may be used. In such embodiments, haptic effects maybe output to one or more users to indicate that action on their part isnecessary to complete the sequence. For example, in one embodiment,multiple mechanics may be repairing a large machine. In such a scenario,each user may wear a device, such as device 410 or 412. As userscomplete portions of the repair or need assistance, they may transmitcommands to trigger haptic effects on other users' devices 410, 412. Bydoing so, the next user may be notified that a task is complete or thatthey need to perform some action to continue the repair process. Therecipient of the haptic effect may provide a haptic confirmation back tothe sender of the command to indicate that the recipient will take thenecessary action or some other indication.

Referring now to FIG. 5, FIG. 5 shows a flow diagram of acomputer-implemented method 500 for haptic confirmation of voicecommands according to one embodiment of the present invention. Themethod 500 shown in FIG. 5 will be discussed with reference to thedevice 100 shown in FIG. 1.

The method 500 begins in block 510 when a processor 100 receives speechinformation. For example, the processor 110 may receive speechinformation encoded in a microphone signal from a microphone 135 or itmay receive the speech information from another processor, such as overa Bluetooth connection. After receiving speech information, the methodproceeds to block 520.

At block 520, after receiving the speech information, the processor 100then executes speech recognition software to recognize the speechinformation. If the processor recognizes the speech information, itattempts to determine a command associated with the speech information.If the processor 110 determines a command associated with the speechinformation, the method proceeds to step 530. However, if the processoris unable to recognize the speech information, or is unable to determinea command associated with the speech information, the method proceeds toblock 522.

In block 522, the processor 110 generates a second actuator signalconfigured to cause an actuator to output a second haptic effect. Thesecond haptic effect is configured to indicate to the user that thespeech information was not recognized, or that no command correspondingto the speech information was found. The processor 110 then transmitsthe actuator signal to the actuator 130 as shown in block 524.

In block 530, the processor 110 generates a first actuator signalconfigured to cause an actuator to output a first haptic effect. Thefirst haptic effect is configured to indicate to the user that thespeech information was recognized and that a command corresponding tothe speech information was found. The processor 110 then transmits theactuator signal to the actuator 130 as shown in block 532. Aftertransmitting the first actuator signal to the actuator 130, the methodproceeds to block 534.

In block 534, the processor 110 determines whether additional speechinformation is available. For example, as described above, the processor110 may attempt to recognize parts of the speech information as it isreceived and then provide haptic effects to indicate that parts of thespeech information were recognized. For example, a user may attempt toadd an event to calendar, and the processor may perform steps 510-532 orsteps 510-524 for each component of the event, such as the date, time,and location of the event. If additional speech information is availableto be received, the method returns to step 510 to receive the additionalspeech information. However, if no additional speech information isreceived, the method 500 proceeds to step 536.

At block 536, the processor 110 executes a function associated with therecognized speech information and the command. For example, if thespeech information comprised “call home” and the determined command todial a phone number associated with a contact called “home,” theprocessor would then execute a dialing function to dial the numberassociated with “home.” After the function has begun, the methodproceeds to block 540 which is a test condition for a loop based onwhether the function has completed or not.

In block 540, the processor 110 determines whether the function hascompleted or not. If the function has not completed, the method proceedsto step 542 to output a haptic effect to indicate that the function isstill executing. However, if the function has completed, the methodproceeds to step 550, where another haptic effect is output to indicatethat the function has completed.

Blocks 542 through 546 are steps to output a haptic effect to indicatethat a function is still executing. In block 542, the processor 110generates an actuator signal corresponding to a haptic effect thatindicates that the function is still executing. In one embodiment, theprocessor 110 may generate the same actuator signal regardless of howmany times the haptic effect has been output. For example, if thefunction requires ten minutes to complete and a haptic effect is outputevery 30 seconds, the processor may generate the same actuator signaleach time the haptic effect is output. Thus, a user would feel the samehaptic effect about every 30 seconds. However, in some embodiments,different haptic effects may be output depending on the status of thefunction. For example, in one embodiment, the processor 110 may generatedifferent actuator signals each time the method returns to block 542. Insuch an embodiment, the processor 110 may generate actuator signals togenerate increasingly stronger haptic effects for each iteration throughblocks 542-546. In one embodiment, the processor 110 may generate adifferent actuator signal if the function is still executing, butencounters an error. In such an embodiment, the processor 110 maygenerate an actuator signal associated with a haptic effect thatindicates that the function encountered an error.

After generating an actuator signal in block 542, in block 544 theprocessor 110 transmits the actuator signal to the actuator 130 to causethe actuator to output a haptic effect. In some embodiments, theprocessor may transmit the actuator signal to another device comprisingan actuator. For example, in the system shown in FIG. 2, processor 160may transmit the actuator signal to processor 110. Processor 110 maythen transmit the actuator signal to the actuator 130. As describedabove, processor 160 may transmit a high-level actuator signal toprocessor 110, which may then generate a low-level actuator signal thatthe processor 110 then transmits to the actuator 130. After the actuatorsignal has been transmitted, the method proceeds to block 546.

In block 546, the method delays for a period of time before returning toblock 540. A delay may be advantageous to allow time to pass betweenhaptic effects. For example, a delay may allow haptic effects to beoutput once every 30 seconds so that the user is not feeling hapticeffects repeatedly over very short periods of time, which may bedistracting to the user. However, in some embodiments, the delay timemay be set to 0, which may allow haptic effects to be output as needed.After the delay has elapsed, the method returns to block 540.

After the function has completed, the method proceeds to block 550 atwhich time the processor 110 generates an actuator signal correspondingto a haptic effect that indicates that the function has completed. Forexample, in one embodiment, the processor 110 may generate an actuatorsignal corresponding to a haptic effect that indicates the functioncompleted successfully. Alternatively, the processor 110 may generate anactuator signal corresponding to a haptic effect that indicates thefunction encountered an error and terminated before running tocompletion. After the actuator signal has been generated, the methodproceeds to block 552 where the processor 110 transmits the actuatorsignal to the actuator 130 to cause the actuator 130 to output a hapticeffect. After which, the method returns to block 510 and the processor110 attempts to receive more speech information.

Referring now to FIG. 6, FIG. 6 shows a block diagram of a system 600for haptic confirmation of commands according to one embodiment of thepresent invention. The system 600 comprises a device 610 having ahousing 605. The device 610 further comprises a processor 620, a memory625, an actuator 630, and a sensor 640. The processor 620, the memory,625, the actuator 630, and the sensor are each disposed within orcoupled to the housing 605. The processor 620 is in communication witheach of the memory 625, the actuator 630, and the sensor 640. Theprocessor is further in communication with a display 650, which maycoupled to the housing, disposed within the housing, or a separatedevice. The device 610 is configured to receive commands based on sensorsignals received from the sensor and to output haptic effects based onthe received sensor signals.

In one embodiment the device 610 may be carried or worn by a user andthe sensor 640 may comprise an optical sensor configured to detect alocation a user's eye is viewing. For example, display 650 may comprisea display worn by a user, such as goggles with an integrated display.The processor 620 may cause the display 650 to display images or textcorresponding to commands, such as commands to control a vehicle (e.g. awheelchair) or commands to send to another person, such as an order to asoldier. To issue a command, a user may look at the desired command. Thesensor 640 detects the orientation of the user's eye is looking andtransmits a sensor signal to the processor 620. The processor 620determines a command associated with the orientation and generates anactuator signal to cause the actuator to output a haptic effect. Theprocessor 620 then transmits the actuator signal to the actuator 630. Inanother embodiment, the device 610 further comprises a networkinterface, such as network interface 315 shown in FIG. 4. In such anembodiment, the processor 620 may receive the command from a seconddevice, such as device 412 shown in FIG. 4, across a network 480. Thedevice 610 may output a haptic effect to the user to indicate a commandhas been received and may display possible responses to the command onthe display 650. The user of the device 610 may select a response fromthe available responses to send to the second device 412 across thenetwork. Such an embodiment may be useful in a covert military operationwhere spoken commands or responses may not be desirable.

In a related embodiment, the selected response may be sent to aplurality of recipients. For example, a military unit, such as a squadof soldiers, may be in communication with a command center and providehaptic responses to received voice commands. In such an embodiment, thecommand center may transmit voice or haptic commands to each of thesoldiers in the squad and may receive responses or acknowledgements fromeach soldier. A further embodiment may include multiple command centersto receive responses from each of the soldiers. For example, a commanderof the squad may be in a base camp near the squad's operations or may bewith the squad but separated due to the nature of the operation. Inaddition, other locations may monitor the squad's activities fromanother location, such as on a ship or at another military installation.In such an embodiment, each of the other locations may receivecommunications to and from the squad, including haptic commands andconfirmations from the squad.

Referring again to FIG. 6, one embodiment of the present invention mayoutput haptic effects that emulate the command issued or the functionexecuted. For example, in one embodiment, a device 610 may be configuredto control a vehicle, such as a wheelchair. In such an embodiment, itmay be advantageous to provide haptic effects that emulate commandsissued to the vehicle. For example, if a user selects a command to turnthe vehicle to the right, the processor 620 may generate an actuatorsignal to cause the actuator to output a haptic effect the emulates aright turn. For example, in a wheelchair embodiment, the actuator maycomprise a plurality of actuators. The actuators may then output avibration that begins in the middle of the back of the wheelchair andmoves to the right. For example, the actuators may be placed along ahorizontal line in the back of the wheelchair, such that the actuatorsare activated in succession. The user may thus perceive the vibrationmoving to the right across their back and therefore receive feedbackindicating that the command was recognized.

Referring now to FIG. 7, FIG. 7 shows a flow diagram of acomputer-implemented method for haptic confirmation of commandsaccording to one embodiment of the present invention. FIG. 7 will bedescribed with respect to the system 600 shown in FIG. 6.

In the embodiment shown in FIG. 7, the method 700 begins in block 710when the processor 620 receives a sensor signal from the sensor 640. Forexample, in one embodiment the sensor signal may correspond to a usertouching a location on a touch-sensitive display, such as display 650.In another embodiment, the sensor signal may correspond to anorientation of a user's eye. After receiving the sensor signal, themethod 700 proceeds to block 720.

In block 720, the processor 620 attempts to identify a commandcorresponding to the received sensor signal. For example, the processor620 may identify a button or user interface element at a locationcorresponding to a sensed position on the touch-sensitive displayscreen. If the processor 620 determines a command corresponding toinformation received in the sensor signal, the method 700 proceeds toblock 730. Otherwise, the method 700 proceeds to block 722.

If the method proceeds to block 722, the processor 620 generates anactuator signal corresponding to a haptic effect configured to indicatethat a command was not recognized based on the input received from thesensor. After generating the actuator signal, the method proceeds tostep 724 when the processor transmits the actuator signal to theactuator 630 to cause the actuator to output a haptic effect.

If the method proceeds to block 730, the processor 620 generates anactuator signal corresponding to a haptic effect configured to indicatethat a command was recognized based on the input received from thesensor. After generating the actuator signal, the method proceeds tostep 750 when the processor transmits the actuator signal to theactuator 630 to cause the actuator to output a haptic effect. Afteroutputting the haptic effect, the method proceeds to block 750 theprocessor 610 executes a function associated with the command, though insome embodiments, the processor 610 may begin executing the functionbefore generating the actuator signal or transmitting the actuatorsignal to the actuator.

While the methods and systems herein are described in terms of softwareexecuting on various machines, the methods and systems may also beimplemented as specifically-configured hardware, such afield-programmable gate array (FPGA) specifically to execute the variousmethods. For example, referring again to FIGS. 1-4 and 6, embodimentscan be implemented in digital electronic circuitry, or in computerhardware, firmware, software, or in combination of them. In oneembodiment, a computer may comprise a processor or processors. Theprocessor comprises a computer-readable medium, such as a random accessmemory (RAM) coupled to the processor. The processor executescomputer-executable program instructions stored in memory, such asexecuting one or more computer programs for editing an image. Suchprocessors may comprise a microprocessor, a digital signal processor(DSP), an application-specific integrated circuit (ASIC), fieldprogrammable gate arrays (FPGAs), and state machines. Such processorsmay further comprise programmable electronic devices such as PLCs,programmable interrupt controllers (PICs), programmable logic devices(PLDs), programmable read-only memories (PROMs), electronicallyprogrammable read-only memories (EPROMs or EEPROMs), or other similardevices.

Such processors may comprise, or may be in communication with, media,for example computer-readable media, that may store instructions that,when executed by the processor, can cause the processor to perform thesteps described herein as carried out, or assisted, by a processor.Embodiments of computer-readable media may comprise, but are not limitedto, an electronic, optical, magnetic, or other storage device capable ofproviding a processor, such as the processor in a web server, withcomputer-readable instructions. Other examples of media comprise, butare not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip,ROM, RAM, ASIC, configured processor, all optical media, all magnetictape or other magnetic media, or any other medium from which a computerprocessor can read. The processor, and the processing, described may bein one or more structures, and may be dispersed through one or morestructures. The processor may comprise code for carrying out one or moreof the methods (or parts of methods) described herein.

General

The foregoing description of some embodiments of the invention has beenpresented only for the purpose of illustration and description and isnot intended to be exhaustive or to limit the invention to the preciseforms disclosed. Numerous modifications and adaptations thereof will beapparent to those skilled in the art without departing from the spiritand scope of the invention.

Reference herein to “one embodiment” or “an embodiment” means that aparticular feature, structure, operation, or other characteristicdescribed in connection with the embodiment may be included in at leastone implementation of the invention. The invention is not restricted tothe particular embodiments described as such. The appearance of thephrase “in one embodiment” or “in an embodiment” in various places inthe specification does not necessarily refer to the same embodiment. Anyparticular feature, structure, operation, or other characteristicdescribed in this specification in relation to “one embodiment” may becombined with other features, structures, operations, or othercharacteristics described in respect of any other embodiment.

That which is claimed is:
 1. A system for generating haptic effects toconfirm receipt of a voice command, the system comprising: a microphone;a housing configured to be contacted by a user; an actuator coupled tothe housing, the actuator configured to output a haptic effect; and aprocessor coupled to the actuator and the microphone, the processorconfigured to: receive speech information from the microphone; recognizethe speech information and determine a command associated with thespeech information; generate a first actuator signal configured to causethe actuator to output a first haptic effect; and transmit the firstactuator signal to the actuator.
 2. The system of claim 1, wherein theprocessor is further configured to: execute a function associated withthe speech information; generate a second actuator signal configured tocause the actuator to output a second haptic effect; and transmit thesecond actuator signal to the actuator upon completion of execution ofthe function.
 3. The system of claim 2, wherein the second haptic effectis configured to emulate the function.
 4. The system of claim 3, whereinthe function comprises activating a turn signal indicator and the secondhaptic effect comprises a vibration that travels in the direction of theimpending turn.
 5. The system of claim 1, further comprising a speakerin communication with the processor, and wherein the processor isfurther configured to: generate an audio signal configured to cause thespeaker to output a sound; and transmit the audio signal to the speaker.6. The system of claim 1, further comprising a display in communicationwith the processor, and wherein the processor is further configured to:generate a display signal configured to cause the display to display animage; and transmit the display signal to the display.
 7. The system ofclaim 1, wherein the first haptic effect is configured to emulate thecommand.
 8. The system of claim 1, wherein the microphone, actuator, andprocessor are disposed within the housing.
 9. The system of claim 1,wherein the housing comprises one of a Bluetooth headset, a Bluetoothearbud, a cell phone, a personal digital assistant, a touch-sensitivesurface, a mouse, or a keyboard.
 10. A computer-implemented methodcomprising the steps of: receiving speech information from a microphone;recognizing the speech information and determining a command associatedwith the speech information; generating a first actuator signalconfigured to cause an actuator to output a first haptic effect; andtransmitting the first actuator signal to the actuator;
 11. Thecomputer-implemented method of claim 10, further comprising: executing afunction associated with the command; generating a second actuatorsignal configured to cause the actuator to output a second hapticeffect; and transmitting the second actuator signal to the actuator uponcompletion of execution of the function.
 12. The computer-implementedmethod of claim 11, wherein the second haptic effect is configured toemulate the function.
 13. The computer-implemented method of claim 12,wherein the function comprises activating a turn signal indicator andthe second haptic effect comprises a vibration that travels in thedirection of the impending turn.
 14. The computer-implemented method ofclaim 10, further comprising: generating an audio signal configured tocause a speaker to output a sound; and transmitting the audio signal tothe speaker.
 15. The computer-implemented method of claim 10, furthercomprising: generating a display signal configured to cause a display todisplay an image; and transmitting the display signal to the display.16. The computer-implemented method of claim 10, wherein the firsthaptic effect is configured to emulate the command.